One of the main functions of computer technology is to help people to quickly locate some piece of key information that they need. A vast amount of information is available electronically and it can be very difficult to find a single key piece of information using current search engines.
Instead of search engines, information can be gathered and processed using information agents to retrieve the information. These agents provide a number of ways to retrieve and organize information. Information agents are capable of accessing information from multiple sources, and then filtering information by relevance to a user. The most basic systems use non-cooperating agents to perform an information retrieval task. Enhanced systems use cooperating agents and adaptive information agents that can deal with uncertain, incomplete, or vague information. Information agents can efficiently gather heterogeneous and frequently changing information. Systems using agents are generally distributed systems using multiple computers.
Cluster analysis is a method that has been used to categorize information. Commonly, distributed clustering techniques have maintained some form of centralized information gathered from the distributed set of documents. This information must be updated any time a document is added or removed from the document set, which necessitates significant processing, communication and storage capacity. It would be advantageous to provide a distributed clustering method that does not need any centralized repository of information, so that when a document is added or removed from the system there is very little information that must be updated.
In characterizing documents for clustering purposes, a vector space model (VSM) has been used, where each unique word within a collection of documents represents a dimension in space, while each document represents a vector within that multidimensional space. Vectors that are close together in this multidimensional space form clusters, or groups of documents that are similar.
The use of information agents to collect and display information using similarity and clustering methods is described and illustrated in Potok et al. U.S. Pat. Pub. No. US 2003/01207639, published Jun. 26, 2003.
Once the information is collected and parsed into documents, there is a need for effective organization and storage of documents for purposes of search, retrieval, display and evaluation of the documents.